Media systems transmit media data, such as video data, over wired and/or wireless channels. Data transmitted over such channels may be lost or corrupted or may experience delays along the way, perhaps arriving late at its destination. Late or lost data can be particularly troublesome for video data that are predictively encoded (compressed) using techniques such as but not limited to MPEG (Moving Pictures Experts Group) encoding. Predictive encoding introduces dependencies in the encoded data, so that the decoding of some data depends on the decoding of other data. While predictive encoding generally improves the amount of compression, it can also result in error propagation should data relied on for the decoding of other data be lost or arrive late. Any late or lost data can impact the quality of the reconstructed (decoded or decompressed) video data. However, the impact can be aggravated if the lost or late data is part of a reference frame used for motion compensated prediction because errors will propagate to other frames that are dependent on the reference frame.
For instance, consider a moving object that appears in different positions in successive frames of video data. Using predictive encoding techniques, the object is described by data in the first frame, but in the second frame the object is described using a motion vector that describes how the object moved from the first frame to the second frame. Thus, only the data for the motion vector needs to be transmitted in the second frame, improving the amount of compression because the data describing the object does not need to be retransmitted. However, if the motion vector is not received, then the object cannot be properly rendered when the second frame is reconstructed into a video image, thus reducing the quality of the reconstructed video. Subsequent frames in which the object appears may also be affected, because they may depend on proper placement of the object in the second frame.
To alleviate the impact of absent (e.g., missing, lost, late or incorrectly received) data on the quality of the reconstructed video, a video decoder can apply an error recovery (e.g., error concealment) process to the received data. Studies have shown that the quality of the reconstructed video can be significantly improved if motion vectors can be recovered (e.g., estimated). Temporal error concealment improves the quality of the reconstructed video by estimating missing or incorrectly received motion vectors in a current frame using properly received information from the current frame and/or preceding frames. In other words, a goal of temporal error concealment is to estimate motion vectors using their spatial as well as temporal associates.
Conventional temporal error concealment techniques are based in the pixel domain. Consider a frame (the current frame) in which a motion vector associated with an area (e.g., a macroblock of interest) in the frame is missing. A set of motion vectors is formed by selecting motion vectors associated with macroblocks that surround the macroblock of interest in the current frame and motion vectors associated with macroblocks that surround the co-located macroblock in the reference frame (the co-located macroblock is the macroblock that is at the same position in the reference frame as the macroblock of interest is in the current frame). With a pixel-domain approach, a measure of distortion is calculated for each of the motion vectors in the set. To evaluate the distortion, pixel values are taken from the reconstructed frame buffer. In a motion select technique, the motion vector that minimizes the distortion measure is chosen as the replacement for the absent motion vector. In a motion search technique, a search for a motion vector that minimizes the distortion measure is performed within, for example, a 3×3 window of macroblocks.
Pixel-domain error concealment is problematic because it is computationally complex and time-consuming. Evaluating the distortion for each potential motion vector can require a large number of computations, consuming computational resources and causing delays in the decoding process. Pixel-domain error concealment is most effective when performed after the decoder has finished decoding a frame; hence, the delay introduced by error concealment in the pixel domain may be equivalent to one frame duration. Furthermore, accessing the reconstructed frame buffer to retrieve pixel values for the distortion evaluation takes time, which adds to the delays.
Accordingly, a method and/or system that can reduce computational complexity and decoding delays would be desirable.